Apparatus and method for non-sequential image data transmission and display

ABSTRACT

An apparatus and method for image display and transmission, including display and transmission of data over the Internet. Image transmission is effected by non-sequential streaming of sequential images or of multiple views of a scene or object. The present invention sends image frames from server to browser out of order. Wherever the cursor is positioned, the browser displays the nearest frame which is received from the server. The viewer, therefore, is able to see more sides of the image without waiting for the entire image to be transmitted.

This application claims benefit of Provisional application Ser. No.60/065,433, filed Oct. 30, 1997.

FIELD OF THE INVENTION

The present invention relates to an apparatus and method for image datatransmision and display, including image display and transmission ofimage data over the Internet.

BACKGROUND OF THE INVENTION

In many applications currently utilized within the art, it is common forgraphical or visual data to be transmitted to a user overtelecommunications lines such as phone lines. Such image transmissionsinclude transmissions of video images, images of a scene, of a threedimensional object, and so forth. Particularly in low bandwidthenvironments, image transmission and display can be extremely slow.Accordingly, it is a goal of the present invention to address certain ofthe drawbacks of the transmission systems used in the present state ofthe art, while particularly taking into account current bandwidthlimitations.

SUMMARY OF THE INVENTION

In accordance with the present invention, an apparatus and method isprovided for image display and transmission, including display andtransmission of image data over the Internet. Image transmission iseffected by nonsequential streaming of sequential images, or or ofmultiple views of a scene or object. In accordance therewith, rotationof objects or scenes and viewing of multiple perspectives can beaccomplished more rapidly, since views can be preliminarily displayedbefore completion of data streaming (which itself is limited in speeddue to the low bandwidth environment). Accordingly, the presentinvention is particularly useful for transmission of image data over theInternet.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a first perspective view of an image for transmission inaccordance with the present invention.

FIG. 2 is a second perspective view of the image of FIG. 1, from adifferent angle of view than that shown in FIG. 1.

FIG. 3 is a third perspective view of the image of FIG. 1, from adifferent angle of view than those shown in FIGS. 1 and 2.

FIG. 4 is a table illustrating a sample embodiment of the nonsequentialstreaming of multiple frames, in accordance with the present invention.

FIG. 5 is a second embodiment of the nonsequential streaming disclosedherein.

FIG. 6 is a first perspective view of an image transmitted and displayedto the viewer in accordance with the present invention.

FIG. 7 is a second perspective view of the image of FIG. 1, displayed tothe viewer when the cursor is located in the shown position.

FIG. 8 is a third perspective view of the image of FIG. 1, displayed tothe viewer when the cursor is located in the shown position.

FIG. 9 is a third perspective view of the image of FIG. 1, displayed tothe viewer when the cursor is located in the shown position.

DETAILED DESCRIPTION OF THE INVENTION AND THE PREFERRED EMBODIMENTS

In accordance with the present invention, a streaming and image displaytechnology is provided for display of images transmitted overtelecommunications lines, including phone lines or even satellitetransmissions. The invention is particularly suited for low-bandwidthenvironments such as the Internet. The present invention (also calledTrueMotion VR™ or TMVR™ by the present inventors), allows a personviewing a web page (“the viewer”) to select and move among the images ina streamed video sequence before all the frames of that video sequencehave been transmitted to the viewer. Software based on this technologytherefore accepts a stream of video images, and begins displaying thoseimages before receiving all of them. The viewer can then, for example,rotate an object early on during transmission without waiting for thefull transmission to be completed. In this way, the viewer can quicklymove a cursor over the TrueMotion VR™ image to quickly view an objectand gain a better understanding of its shape, color and form in time andspace.

In the preferred embodiment, the invention is implemented in software asa Netscape™ client plugin or ActiveX control program; although,alternatively, the technology described herein can be applied in manydifferent environments. In the preferred environment, the programresides within an Internet/Intranet HTML page and displays image framesdepending on where the viewer of these images has placed a cursor insidethe image on his or her computer screen. This navigation techniqueallows the content developer to produce images of an object that can bedisplayed based on the location of a cursor within the image.

The simplest content architecture consists of a series of frames shotfrom different angles of an object, as shown for example in FIGS. 1-3.The object can be made to appear to rotate from side to side if theviewer moves the cursor right and left. As the cursor moves to theright, the plugin displays images from camera angles progressively tothe left so the object appears to rotate to the right. As the cursormoves to the left, the plugin displays images from camera anglesprogressively to the right so that the object appears to rotate to theleft. The images shown in FIGS. 1 through 3, when viewed sequentially,show the cursor moving to the right and the object appearing to rotateto the right.

A more complex content architecture consists of several series of framescalled “rows” or “rings”. As the cursor moves up and down over the faceof the plugin, it displays images from different rings. Depending on thecontent of the different rings, the image could “zoom” to become largerand smaller as the cursor moves up and down. Alternatively, the cameracould appear to move up over the object. Simply changing the way theimage is built can allow the content developer to make the image appearto move closer and farther, or to rotate up and down.

In accordance with the present invention, a method for data transmissionto the user is provided, which is based on a unique approach tostreaming video. A video clip is normally comprised of a linearcollection of images (or frames) arranged in time order (i.e. asequential stream of image frames). Video streaming software transmitsthe clip from one computer to another. In the method of the prior art,the software transmits the frames of the clip in the same order in whichthe clip was created. A video player, as it plays these frames in order,gives the impression of coherent motion because frames that are earlierin time are displayed before those that are later in time.

In what the present inventors call ordered streaming, frames are sentfrom the server computer to the client computer (which is running abrowser) in the order that they will be displayed. Frame 1 is followedby Frame 2, Frame 2 is followed by Frame 3, and so forth. For example,with reference to the figures, FIG. 1 would be transmitted first,followed by FIG. 2, then by FIG. 3.

In low bandwidth situations, when it takes a significant length of timeto transmit each frame, ordered streaming is not considered by thepresent inventors to be a sufficiently effective strategy. Thetransmission medium, typically a fairly slow, analog telephone line,gets clogged up with frames that are similar to one another. Yet theviewer ultimately wants the ability to see all sides of the object asquickly as possible. In accordance with the present invention, it isbelieved that the viewer would be willing to sacrifice some angularresolution and thus smoothness of motion as the clip of images begins tostream into the browsers.

Accordingly, in the method and software of the present invention,TrueMotion VR™ sends the frames from server to browser in an out oforder sequence (i.e. as a non-sequential stream of image frames). Anynumber of frames can be so transmitted, whether 5, 10, 20, 50, 100,1000, 10,000, or more frames. Wherever the cursor is positioned, thebrowser displays the nearest frame that it has received from the server(i.e. the proximal image). When all the frames are received, the browsercan display the correct frame; when frames are still arriving, thebrowser displays the proximal image, i.e. the closest frame that it can.The software modifications required for modification of the knownsoftware for conducting transmission and display of images on screen inaccordance with the methods of the present invention are well within theskill of one of ordinary skill in the art.

One example of such an out-of-order strategy for 24 frames in the AVIfile and one row, reorders the frames from line to scattered. Thisstrategy is shown in FIG. 4. Frames 1 through 24 (as shown on the upperline of the figure) represents the in sequence, or linear order of thecomplete series of images, while Frames 1, 13, 7, 20, 4, etc. representsthe out of sequence order that those images are transmitted to the user.If the Frames 1 through 24, for example, represent views of an objecttaken at successive angles, the result for the viewer is that all sidesof the object can be seen quickly and, as the viewer waits, the angularresolution between “steps” improves.

As an example, we can examine the behavior of the plugin program afterfour frames have been received by the browser. With an ordered stream,Frames 1, 2, 3 and 4 would be transmitted to the viewer in sequence andthe plugin would display:

frame 1 if the cursor was in the left hand 24th of the window (position1),

frame 2 if the cursor was in the second 24th (position 2),

frame 3 if the cursor was in the third 24th,

frame 4 if the cursor was in the next ⅞ of the screen.

Most of the time, frame four would be displayed. Eventually all of theframes would be transmitted to the user. In the meantime, however,during the transmission period, the overall viewable image would belimited, as shown in the above example.

With a scattered stream, on the other hand, four out of sequence frameswould be transmitted to the user, e.g. Frames 1, 13, 7 and 20, in thatorder, as shown in FIG. 4. After transmission of the four out ofsequence frames, a broad range of angles of the object would beviewable. Specifically, the plugin would display:

frame 1 if the cursor was in positions 1-4, as shown in FIG. 6;

frame 7 if the cursor was in positions 5-10, as shown in FIG. 8;

frame 13 if the cursor was in positions 11-16, as shown in FIG. 7;

frame 20 if the cursor was in positions 17-24, as shown in FIG. 9.

This scattered stream display methodology, therefore, allows the viewerto see more of the object sooner. Although the full series of frames hasnot yet been transmitted, the viewer nonetheless has a broader spectrumof views available early on than was provided with the ordered streamtransmission.

Many other re-ordering strategies are possible. The scattered streamprovided above is just one example of a particularly useful scatteringof the frames. Another very useful scheme arranges the sequential streamof image frames into subsets of image frames or “slices”. In thisexample, illustrated in FIG. 5, four slices are sent—the first frame ofeach slice, followed by the second frame of each slice, and so forth.For Frames 1-24, the first slice includes Frames 1-6, the second sliceincludes Frames 7-12, the third slice includes Frames 13-18, and thefourth slice includes Frames 19-24. Thus, the first four framestransmitted to the user correspond to the first frames of each of theslices, or Frames 1, 7, 13, and 19. Likewise, the next four framestransmitted to the user correspond to the second frames of each of theslices, or Frames 2, 8, 14, and 20. The remaining frames are transmittedin similar fashion. In other words, the linear sequence of images issubdivided into some number of adjacent subsequences or “slices”. Afirst frame is sent from each subsequence or slice, then a second framefrom each slice, and so on.

Finally, custom orderings by the content developer can allowparticularly interesting viewing angles to be sent early in the stream.

Having described this invention with regard to specific embodiments, itis to be understood that the description is not meant as a limitationsince further modifications and variations may be apparent or maysuggest themselves to those skilled in the art. It is intended that thepresent application cover all such modifications and variations.

We claim:
 1. A method for transmission of image data overtelecommunications lines, comprising the steps of: (a) providing imagedata, said image data comprising a sequential stream of image frames;(b) selecting a first image frame out of said sequential stream of imageframes; (c) selecting further image frames from said sequential streamof image frames to form a non-sequential stream of image frames fortransmission to a viewer, wherein the majority of the image framestransmitted in said non-sequential stream of image frames arenon-sequential to the previously transmitted image; (d) transmittingsaid non-sequential stream of image frames over telecommunicationslines; (e) selecting a proximal image from said non-sequential stream ofimage frames transmitted to the user, said proximal image being theimage nearest to a frame position selected by the viewer, and (f)displaying said proximal image on a viewer's terminal.
 2. A method asclaimed in claim 1, wherein said non-sequential stream of image framescomprises at least 5 frames.
 3. A method as claimed in claim 1, whereinsaid non-sequential stream of image frames comprises at least 10 frames.4. A method as claimed in claim 1, wherein said non-sequential stream ofimage frames comprises at least 20 frames.
 5. A method as claimed inclaim 1, wherein said non-sequential stream of image frames comprises atleast 50 frames.
 6. A method as claimed in claim 1, wherein saidnon-sequential stream of image frames comprises at least 100 frames. 7.A method as claimed in claim 1, wherein said non-sequential stream ofimage frames comprises at least 1000 frames.
 8. A method as claimed inclaim 1, wherein said non-sequential stream of image frames comprises atleast 10,000 frames.
 9. A method as claimed in claim 1, wherein saidnon-sequential stream of image frames is transmitted through a lowbandwidth medium.
 10. A method as claimed in claim 1, wherein the firstimage frame to be transmitted of said non-sequential stream of imageframes is the first image frame in said sequential stream of imageframes.
 11. A method as claimed in claim 10, wherein the second imageframe to be transmitted of said non-sequential stream of image frames isthe last image frame in said sequential stream of image frames.
 12. Amethod as claimed in claim 1, wherein said sequential stream of imageframes corresponds to a sequential series of views of a threedimensional object.
 13. A method as claimed in claim 1, wherein saidsequential stream of image frames corresponds to a sequence of video.14. A method as claimed in claim 1, wherein said telecommunicationslines are telephone lines, and wherein said image data is transmitted toa viewer over the Internet.
 15. A method as claimed in claim 1, whereinsaid sequential stream of image frames is divided into slices, each ofsaid slices being a subset of said sequential stream of image frames,and further comprising the step of transmitting a frame from each ofsaid slices in said non-sequential stream of image frames beforetransmitting a second frame from any of said slices.
 16. An apparatusfor transmission and display of an image, presented as a sequentialstream of image frames, transmitted through a low bandwidth medium,comprising: (a) means for non-sequentially selecting a series of imagesframe out of a sequential stream of image frames, (b) means fortransmitting said non-sequentially selected image frames to the viewer;(c) means for locating one of said non-sequentially selected andtransmitted images frames nearest to a frame position selected by theviewer; and, (d) means for displaying said one located frame on theviewer's terminal.
 17. A method for transmission and display of animage, presented as a sequential stream of image frames, transmittedthrough a low bandwidth medium, comprising the steps of: (a)non-sequentially selecting a series of images frame out of a sequentialstream of image frames, (b) transmitting said non-sequentially selectedimage frames to the viewer; (c) locating one of said non-sequentiallyselected and transmitted images frames nearest to a frame positionselected by the viewer; and, (d) displaying said one located frame onthe viewer's terminal.